Skip to content

test(prose): give the eight read-only cases something that can fail - #565

Merged
leeovery merged 1 commit into
mainfrom
test/prose-claims-that-can-fail
Jul 27, 2026
Merged

test(prose): give the eight read-only cases something that can fail#565
leeovery merged 1 commit into
mainfrom
test/prose-claims-that-can-fail

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Eight cases were enforcing nothing, all showing green. Every entry case declared exactly one check — engine_before_write — and those walks write nothing, so it reported "no workflow state was written" and passed regardless of what the prose did.
  • A check with nothing to examine now reports N/A, not PASS. A green tick for absence of coverage is precisely how this went unnoticed.
  • The eight get real declarations, drawn from claims they already make. Their "further claims" turn out to be not-called assertions almost word for word — "the entry skill records nothing", "an entry that only validates writes nothing", "the knowledge base is queried, never written to" — so each becomes a calls_exclude over topic start / topic reopen / manifest set / knowledge index. What each says it reads becomes a calls_include over the gate it must render.
  • calls_in_order is new — the one thing presence could never settle. A gate read after the arm it was supposed to select proves the arm was chosen some other way. Declared commands must appear as a subsequence, anything permitted between them.
  • P9 corrected. It demanded every case run until state landed. Wrong twice: it treated file changes as the only hard evidence when recorded actions are equally deterministic, and it would have pushed short entry walks into needing conversation stubs — trading live prose for hand-written fiction. A handoff is a fine ending. The rule is that every claim must have a deterministic answer; the stop point follows.

Verified against live walks

These declarations were derived from each case's claims and its skill's actual calls, then confirmed by running the whole corpus: all eight passed on claude-sonnet-5, so the gates, orderings and not-called assertions were right as written. No corrections were needed.

Test plan

  • Prose suites 75/75, including new coverage for ordering (in-sequence, out-of-sequence, gaps tolerated, missing call, minimum length) and for N/A vs PASS
  • Corpus valid
  • Full run: 8/8 PASS

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565 👈 current
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from fd3126f to c4b2589 Compare July 26, 2026 19:39
This was referenced Jul 26, 2026
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from ca31778 to 4a23bdc Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 76755ba to ce1872c Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 4a23bdc to 4b9b166 Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from ce1872c to b1f3c27 Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 4b9b166 to fc39dec Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from b1f3c27 to 38d45b8 Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from fc39dec to 60d545f Compare July 27, 2026 15:42
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 38d45b8 to e598cf0 Compare July 27, 2026 15:42
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 60d545f to 6752084 Compare July 27, 2026 15:44
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from e598cf0 to 1864e37 Compare July 27, 2026 15:44
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 6752084 to 66e9b2a Compare July 27, 2026 15:45
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 1864e37 to 9d29328 Compare July 27, 2026 15:45
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 66e9b2a to c811de2 Compare July 27, 2026 15:47
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 9d29328 to 404e386 Compare July 27, 2026 15:47
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from c811de2 to f41095d Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 404e386 to 0c0f5c4 Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from f41095d to e2a0c10 Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 0c0f5c4 to 09b6afe Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from e2a0c10 to ea30730 Compare July 27, 2026 15:51
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 09b6afe to 967e8d3 Compare July 27, 2026 15:51
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from ea30730 to 5ff2f71 Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 967e8d3 to 70e8adc Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the feat/prose-walker-on-sonnet branch from 5ff2f71 to 8c4377a Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 70e8adc to 1041d1c Compare July 27, 2026 15:54
@leeovery
leeovery changed the base branch from feat/prose-walker-on-sonnet to main July 27, 2026 15:55
Every entry case declared exactly one check, `engine_before_write`, and
those walks write nothing — so it reported "no workflow state was
written" and passed, every time, whatever the prose did. Eight cases with
nothing enforceable behind them, all showing green.

Two changes so that cannot recur quietly.

A check with nothing to examine now reports N/A rather than PASS. A green
tick for absence of coverage is how this went unnoticed in the first
place; the distinction is now on the page.

And the eight get real declarations, drawn from the claims they already
make. Their "further claims" turn out to be not-called assertions almost
word for word — "the entry skill records nothing", "an entry that only
validates writes nothing", "the knowledge base is queried, never written
to" — so each becomes a calls_exclude over `topic start`, `topic reopen`,
`manifest set`, `knowledge index`. What each case says it reads becomes a
calls_include over the gate it must render.

calls_in_order is new, and is the one thing presence could never settle:
a gate read *after* the arm it was supposed to select proves the arm was
chosen some other way. Declared commands must appear as a subsequence,
with anything permitted in between.

P9 is corrected too. It demanded every case run until state landed, which
was wrong twice: it treated file changes as the only hard evidence, when
recorded actions are equally deterministic, and it would have pushed
short entry walks into needing conversation stubs — trading live prose
for hand-written fiction. A handoff is a fine ending. The rule is that
every claim must have a deterministic answer; the stop point follows from
that rather than being legislated.

The declarations are derived from the prose and not yet from a live run.
Any that prove wrong get corrected on this branch before it lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from 1041d1c to b28ba29 Compare July 27, 2026 15:55
@leeovery
leeovery merged commit 9551309 into main Jul 27, 2026
@leeovery
leeovery deleted the test/prose-claims-that-can-fail branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant